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Image Sensor For Digital Cameras 



Field of the Invention 

The present invention relates to digital cameras, and more particularly, to an improved 
design for a color sensor array for use in digital cameras and the like. 

Background of the Invention 

A digital color image usually consists of an array of pixel values representing the 
intensity of the image at each point on a regular grid. Typically, three colors are used to 
generate the image. At each point on the grid the intensity of each of these colors is 
specified, thereby specifying both the intensity and color of the image at that grid point. 

Conventional color photography records the relevant image data by utilizing three 
overlapping color sensing layers having sensitivities in different regions of the spectrum 
(usually red, green, and blue). Digital cameras, in contrast, typically utilize one array of 
sensors in a single "layer". 

When only one sensor array is used to detect color images, only one color may be 
detected at any given sensor location. As a result, these sensors do not produce a color image 
in the traditional sense, but rather a collection of individual color samples, which depend 
upon the assignment of color filters to individual sensors. This assignment is referred to as 
the color filter array (CFA) or the color mosaic pattern. To produce a true color image, with a 
full set of color samples (usually red, green and blue) at each sampling location, a substantial 
amount of computation is required to estimate the missing information, since only a single - 
color was originally sensed at each location in the array. 

There are a wide variety of approaches to the reconstruction problem, which is 
frequently referred to as the "demosaicing" task. The performance of all such algorithms 
depends upon the nature of the underlying CFA pattern. One such pattern is taught in U.S. 
Patent 3,971,065. This pattern is generated by a repeating 2x2 kernel, containing two green 
sensors, one red sensor and one blue sensor, as illustrated below: 
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It should be noted that this pattern has twice as many sensors in the green region of 
the spectrum as it does in the blue or red regions of the spectrum. The proponents of this 
5 pattern justify this choice on the grounds that the Human Visual System (HVS) is more 

sensitive to green. In addition, the proponents of this scheme point to the fact that it leads to 
relatively simple demosaicing algorithms. Specifically, if one color channel is sampled more 
densely than the rest, it can be interpolated most easily and then used to guide the 
interpolation of the less densely sampled color channels. 

go 

fi The argument that the HVS is more sensitive to the green portion of the spectrum is 

~jp fundamentally flawed when applied to digital camera systems. The reason the human eye can 
□ afford to have more green cones than red cones, and many fewer blue cones, is that the eye's 

optics effectively low-pass filter the red and blue parts of the spectrum more severely than the 
H 5 green. Hence, the eye cannot effectively utilize additional samples in the red and blue regions 
M= of the spectrum. In effect, the eye has poor spatial resolution for images in the red and blue 
g regions of the spectrum. 

Digital cameras, in contrast, do not suffer from this limitation; hence, there is no 
20 logical reason for limiting the sampling density at any wavelength. If the sampling density in 
any of the color channels is reduced, there will be an increased ambiguity between spatial 
intensity variations and color composition of the original scene. Since such ambiguities are 
to be avoided, sampling densities in each of the colors should be the same. 

25 Another problem with the small 2x2 kernel described above lies in its inability to 

discern color and spatial variations on image components whose spatial dimensions are small 
compared to the kernel's size. Consider a narrow red line (less than one sensor in width) 
running vertically through the image. If the line is positioned such that it is over the column 
of sensors having only blue and green sensors, the system will have difficulty detecting the 
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line. In fact, if neither the blue or green sensors have at least some response in the red region 
of the spectrum, the line cannot be detected at all. Needless to say, accurately determining 
the color of this line would be difficult. 

Small kernels also can present problems in demosaicing scenes having textures. 
Consider an image feature that is on the order of the size of the kernel. Demosaicing 
algorithms have difficulty determining whether the resultant sensor pattern corresponds to a 
smooth area in the scene with saturated color or a textured region having rapidly varying 
intensity of a more neutral color. 

Hence, the size of the kernel should be relatively large, to assure that scene textures do 
not give the appearance of color variations. It should be noted, however, that there is a limit 
on the size of the kernel. As the kernel size grows, the implementation cost of the 
demosaicing algorithm typically increases. Hence, a tradeoff between cost and image quality 
is generally implied. 

It should also be noted that images of interest to humans often include linear edges as 
well as lines. Hence, the sensor pattern should be designed so as to ensure that any linear 
feature in the image always passes through sensors of every color type. Furthermore, the 
distance along the linear feature between color sensors of the same type should be as small as 
possible. Failure to satisfy this requirement results in color ambiguity at the edges of objects 
in the image. 

The 2x2 pattern described above is clearly a poor choice in light of these 
requirements. First, it has a very small kernel. Second, it undersamples two of the color : '~ 
channels. Third, the red and blue sensors are never adjacent to sensors of the same color. 
Finally, horizontal and vertical linear features never pass through all three of the sensor types. 

One prior art solution to the above problems utilizes large kernels containing pseudo- 
random distributions of the different sensor colors to minimize the likelihood that textures 
will induce colors in the demosaiced image. However, such pseudo-random patterns 
introduce other problems. In addition to increasing the computational costs of demosaicing 
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the image, there are regions in which there is too great a distance between color sensors of the 
same color in a region of the image. This can make it difficult for the demosaicing algorithm 
to distinguish between spatial intensity variations and color variations when it attempts to 
reconstruct a full color image. 

Broadly, it is the object of the present invention to provide an improved color image 
sensor array. 

It is a further object of the present invention to provide an image sensor based on a 
kernel that is larger than the 2x2 kernel described above. 

It is yet another object of the present invention to provide an image sensor in which 
linear features pass through sensors for at least three different colors, whose spectral 
responses are linearly independent. 

It is a still further object of the present invention to provide an image sensor in which 
color sensors for the same color are located adjacent to one another to minimize ambiguities 
introduced by rapidly changing intensity patterns in the image. 

These and other objects of the present invention will become apparent to those skilled 
in the art from the following detailed description of the invention and the accompanying 
drawings. 

Summary of the Invention 

The present invention is an apparatus for recording an image. The apparatus includes 
a two-dimensional array of image sensors. Each image sensor provides a measurement of the 
light intensity in a selected spectral region. The two-dimensional array is generated from a 
plurality of identical blocks of sensors, the blocks being juxtaposed to form the array. Each 
of the blocks has equal numbers of sensors for each of the spectral regions, the number of 
different spectral regions being at least three. The sensors in the blocks are arranged in a two- 
dimensional array having a plurality of rows and columns. The sensors in the blocks are 
arranged such that any straight line passing through a given sensor also passes through 
sensors of at least three different colors whose spectral responses are all linearly independent. 
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In the preferred embodiment of the present invention, all three such sensors lie within a disk, 
centered at the first sensor and having a radius no larger than five times the center to center 
spacing of the blocks in the sensor array. In the preferred embodiment of the present 
invention, each of the sensors corresponding to one of the selected spectral regions in blocks 
5 not adjacent to an edge of the two-dimensional array is adjacent to a sensor corresponding to 
the same spectral region. 

Brief Description of the Drawings 

10 Figure 1 is a top view of an image sensor according to the present invention. 

Figure 2 is a top view of another embodiment of an image sensor according to the 
present invention. 



yi 



Figure 3 is a top view of a third embodiment of an image sensor according to the 
present invention. 

Detailed Description of the Invention 



7-0 Refer now to Figure 1 which is a top view of a color image sensdr 10 according to the 

present invention that is based on a 4 x 3 replicating kernel, consisting of horizontally shifted 

§== rows of alternating red, green and blue sensors, as depicted at 12. 

W 
G 

The CFA pattern utilized in sensor 1 0 is a zigzag pattern in which each of the red, 
25 green and blue channels may be traced in a zigzag fashion from the top of the array to the 
bottom. A portion of the zigzag pattern for one set of green sensors is shown at 13. The 
zigzag nature ensures that linear image features cannot pass through a subset of the sensors, 
which excludes one or more of the color channels. 7" ' 

30 It should also be noted that a sensor for one of the colors is adjacent to two other 

sensors of that color throughout the CFA for pixels away from the edges of the array. This 
feature of the pattern makes it easier for the demosaicing algorithm to determine if a rapidly 
changing feature is the result of a change in color or a change in intensity (i.e., a texture). 
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Other members of the same class of zigzag patterns may be generated by adjusting the 
size and arrangement of the rows of the kernel to alter the nature of the zigzag. For example, 
the pattern might shift to the right for five consecutive rows, before it reverses and shifts to 
the left for the next five rows and so on. A sensor with this pattern is shown in Figure 2 at 
5 20. Sensor 20 is constructed from a 3x8 kernel shown at 22. The zigzag pattern has linear 
segments that extend for 5 pixels before changing direction as shown at 23. 

In general, the pattern need not be a zigzag and the number of color sensors is not 
restricted to three. For example, a sensor according to the present invention based on a 
10 CMYG (cyan, magenta, yellow, and green) color scheme is shown in Figure 3 at 30. The 

kernel in this case is a 3x4-pixel array shown at 32. This pattern shifts to the right for 3 
u consecutive rows, and then jumps two rows to the left to repeat itself The zigzag patterns 
Jrf and this latter pattern have in common the property that they are all generated by cyclic 
Ul permutations of a row containing equal numbers of all color sensors. It will be obvious to 
ffj5 those skilled in the art from the preceding discussion that the same strategy may be applied to 
ff a column. 

s 

M= These patterns have 4 beneficial properties. First, the pattern is generated by 

Li 

Ui replicating a relatively small kernel. This simplifies the mathematical operations involved in 
Bf 0 demosaicing algorithms based on linear models that take into account various distortions 
introduced by lens, color responses, etc. 



Second, as any linear boundary in the image is traversed, the boundary repeatedly 
passes through color sensors of at least three linearly independent colors over a relatively 

25 short distance along the length of the boundary. This provides the data needed to distinguish 
color variations from textural variations along the boundary. Since the human eye is 
particularly sensitive to linear edges, this feature provides substantial benefits. The distance 
that must be traversed along the linear boundary before passing over pixels of all of the 
different colors depends on the orientation of the linear boundary and the particular kernel. In 

30 the preferred embodiment of the present invention, the kernel is chosen such that any line 

which passes through the center of a disk whose radius is five times the distance between the 
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centers of two adjacent kernels will pass through at least three colors whose spectral 
responses are linearly independent within that disk. 

Third, within the body of the pattern, each pixel of any given color is adjacent to other 
pixels of that color. This provides the data needed to distinguish color variations from 
textural variations within more general images. 

Fourth, the kernel has equal numbers of color sensors of each color. This assures that 
all colors are sampled with the same density in the resulting CFA pattern. This, in turn, helps 
to ensure good image quality over a wide range of different scene illumination conditions, 
from incandescent lighting through to illuminants that are heavily biased toward the blue end 
of the spectrum. 

The above embodiments of the present invention have been described in terms of 
sensors for specific portions of the spectrum. However, it will be apparent to those skilled in 
the art from the preceding discussion that other choices of spectral regions may be utilized 
without deviating from the teachings of the present invention. 

Various modifications to the present invention will become apparent to those skilled 
in the art from the foregoing description and accompanying drawings. Accordingly, the 
present invention is to be limited solely by the scope of the following claims. 



